FE-1604: Allow Voice interruption by speaking - #9619

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
hash	Ready	Preview	Sep 10, 2026 2:48pm UTC
hashdotdesign-tokens	Ready	Preview	Sep 10, 2026 2:48pm UTC
petrinaut	Ready	Preview	Sep 10, 2026 2:48pm UTC
petrinaut-docs	Ready	Preview	Sep 10, 2026 2:48pm UTC

PR Summary

Medium Risk
Changes real-time voice turn-taking, transcript admission, and Brunch submission ordering; misclassification or race bugs could drop or duplicate answers, though behavior is heavily tested and defaults are conservative.

Overview
Voice can now interrupt assistant playback by speaking, with Interruption by speaking on by default and stored in the browser. While enabled, the mic stays open during canonical speech; speech_started immediately cancels generation and clears output audio without clearing the input buffer. Completed transcripts still go through Brunch once, can be retained until the prior turn settles, and are classified (prompt-regurgitation / self-echo) only for interruption-originated completions against active playback text.

Disabling the preference restores half-duplex behavior: Your turn handoff, mic closed during output, and stricter overlap rules for in-flight input. The Realtime session tracks speaking/stopped input items, defers canonical playback until speech ends, and settles cancelled speech requests more explicitly. The bridge adds ordered input draining, pending-interruption slots, and content-free transcription rejection diagnostics.

UI exposes the preference via voice controls (persisted with readInterruptionBySpeakingPreference / saveInterruptionBySpeakingPreference), new dock notices for pending answers, and Semantic VAD medium eagerness. Docs and a @hashintel/petrinaut patch changeset accompany broad integration/unit test coverage.

^{Reviewed by Cursor Bugbot for commit 4e38d31. Bugbot is set up for automated code reviews on this repo. Configure here.}

I found four behavioral blockers and one mission-contract gap:

1. Short exact self-echo is admitted. classify-interruption.ts returns null for every transcript under six tokens before comparing canonical playback. An exact echo such as “Who approves this?” therefore becomes an answer. The contract permits short novel answers, not short exact echoes.

2. Generated audio awaiting playback escapes spoken interruption. #interruptOutputBySpeaking() cancels only active or already-speaking responses. After response.done but before output_audio_buffer.started, speech does not cancel/clear the terminal buffered response. The manual-cancellation suite models this phase, but the spoken-interruption “generated” case starts playback first.

3. Pause or host Stop silently drops a retained answer. VoiceTurnController.pause() / host Stop call cancelPendingSpeech(), which sets RealtimeBrunchBridge.#pendingInterruption = null. The controller can continue showing “Answer captured. Waiting for Brunch” after the answer has become undeliverable.

4. Cancellation before response.created can leave permanent output ownership. The cancelled-created path returns before recording speechRequestId; its terminal event cannot clear #pendingSpeechRequestIds in the bridge. Later ordinary speech can remain classified against phantom pending output. The bridge test fabricates a terminal speechRequestId that the real session does not emit in this path.

Separately, MISSION.next.md requires every later Voice mission to exercise the typed-origin + Voice-origin + durably-stopped second-tab reopen scenario. FE-1604 exercises Voice but does not include that gate. This is also still deferred in FE-1573 Step B, so the ownership/disposition should be explicit rather than silently left there.

The architecture and overall branch scope otherwise align well: provider logic remains website-owned, Petrinaut stays provider-neutral, and admission still uses the existing Brunch path.

Addressed and pushed in 268303c8a4, with the ownership follow-up in a16c19ef0f (mission authority changes remain committed separately):

1. Exact normalized playback echoes are now rejected before the six-token fuzzy-overlap threshold. The regression also proves a short novel answer is still admitted unchanged.
2. Spoken interruption clears completed canonical audio buffered between response.done and output_audio_buffer.started; response.cancel is sent only while the response is active. The buffer-clear acknowledgement now carries speechRequestId downstream, so both the bridge and controller release generated-but-unplayed output ownership and restore capture.
3. Output cancellation no longer deletes a finalized retained interruption. Delivery waits for both cancellation handoff and chat readiness. Both Stop/cancellation race orders now assert that the first gate alone cannot drain the retained answer.
4. Response-to-speech correlation is recorded as soon as canonical response.created arrives and retained independently through output cleanup, so a pre-created cancellation's terminal event carries speechRequestId and releases bridge ownership.
5. MISSION.md owns the combined continuity gate. voice-history-continuity.integration.test.tsx exercises typed history, supported Voice client-tool provenance, durable composer Stop, close/reopen through a second mounted Flue observation, stopped rendering, and local End Voice without another durable Stop. Per the owner decision, direct-user Voice source reconstruction is explicitly deferred: Flue 2.0.3 exposes no supported caller-metadata/idempotency-key projection, and the prohibited browser-side/text-encoding alternatives were not introduced.

I still see four code-level gaps:

1. High — rejected response creation still leaks output ownership. If spoken interruption marks a pending response.create cancelled and the provider returns correlated conversation_already_has_active_response, openai-realtime-session.ts cancels its local timing and returns without emitting response-terminal or output-interrupted. The bridge and controller therefore retain the canonical-speech-requested ID indefinitely, leaving phantom output ownership until another manual cancellation or reconnect.

2. Medium — Stop can discard an already-completed answer. With start/stop A → start/stop B → completed(B), B waits in #completedInputEvents for A. RealtimeBrunchBridge.cancelPendingSpeech() then calls #retirePendingInputItems(), which rejects and clears B as unavailable even though its transcription is complete. This reintroduces the retained-answer loss when completion is buffered behind an earlier item rather than already moved into #pendingInterruption.

3. Medium — half-duplex playback can still cut off accepted follow-up speech. After start/stop A → start B → completed(A), the controller preserves B and leaves the microphone enabled. If A's canonical response reaches speakCanonical() before B stops, #sendNextCanonicalSpeech() waits for #speakingInputItemId only when interruption mode is enabled; in half-duplex mode it starts playback and marks unfinished B as overlapping, so B's eventual transcript is discarded. Existing regressions stop or complete B before A's response arrives.

4. Medium — pending playback is still classified as an interruption. canonical-speech-requested makes #ownsOutputTurn() true before any output-started event. Speech beginning in that gap records an empty playback snapshot and still runs classifyInterruption(), allowing prompt-regurgitation rejection even though no canonical audio was active. The branch contract says interruption-only classification applies only to input begun during active playback, and the current test explicitly preserves the contrary pending-creation behavior.

One high-severity lifecycle gap remains: cancelled response.create collision handling is incomplete across inverse event orders.

• Collision before interruption: a canonical request emits canonical-speech-requested, receives correlated conversation_already_has_active_response, and is requeued. If the user then speaks before the competing response terminates, #interruptOutputBySpeaking() removes the requeued request and cancels its local timing without emitting response-terminal or output-interrupted. The bridge/controller retain phantom ownership for the original speech request indefinitely.
• Competing terminal before the cancelled request's error: request A is pending and cancelled by speech, request B queues, then the competing response emits response.done before A's correlated collision error. The terminal's resume attempt is blocked by A's still-set #responseCreateEventId; A's error later clears that ID and returns from the cancelled branch without calling #resumeCanonicalSpeechQueue(), so B remains queued permanently even though its segments have already been marked seen upstream.

The new cancelled-creation regression covers speech before the collision error, but not either collision-first cleanup or a queued successor with response.done preceding the correlated error.

One medium-severity lifecycle gap remains: a retained interruption can remain permanently pending after an empty completed response.

In realtime-brunch-bridge.ts, when the active correlated submission reaches ready with a matching completed settlement but no responseSegments, the branch at lines 959–972 clears #activeSubmission, emits canonical-response-ready, and returns without calling #drainPendingInterruption(). If the user’s interrupting answer was retained while that submission was active, no further chat update or playback handoff is guaranteed to drain it. The dock can remain on “Answer captured. Waiting for Brunch.” indefinitely, and later speech is rejected as pending.

The existing empty-response regression submits the next transcript only after settlement. It does not cover retaining the next transcript before the empty settlement completes.

One medium-severity lifecycle gap remains: a retained interruption can remain permanently pending after an empty completed response.

In realtime-brunch-bridge.ts, when the active correlated submission reaches ready with a matching completed settlement but no responseSegments, the branch at lines 959–972 clears #activeSubmission, emits canonical-response-ready, and returns without calling #drainPendingInterruption(). If the user’s interrupting answer was retained while that submission was active, no further chat update or playback handoff is guaranteed to drain it. The dock can remain on “Answer captured. Waiting for Brunch.” indefinitely, and later speech is rejected as pending.

The existing empty-response regression submits the next transcript only after settlement. It does not cover retaining the next transcript before the empty settlement completes.

This issue is already fixed in the latest PR version. The empty-response path now drains retained interruptions, with a regression test covering the exact scenario. Bugbot and tests pass ✅